Continual Reinforcement Learning Formulation For Zero-Sum Game-Based Constrained Optimal Tracking

This study provides a novel reinforcement learning-based optimal tracking control of partially uncertain nonlinear discrete-time (DT) systems with state constraints using zero-sum game (ZSG) formulation. To address optimal tracking, a novel augmented system consisting of tracking error and its integral value, along with an uncertain desired trajectory, is constructed. A barrier function (BF) with a tradeoff factor is incorporated into the cost function to keep the state trajectories to remain within a compact set and to balance safety with optimality. Next, by using the modified value functional, the ZSG formulation is introduced wherein an actor–critic neural network (NN) framework is employed to approximate the value functional, optimal control input, and worst disturbance. The critic NN weights are tuned once at the sample instants and then iteratively within sampling instants. Using control input errors, the actor NN weights are adjusted once a sampling instant. The concurrent learning term in the critic weight tuning law overcomes the need for the persistency excitation (PE) condition. Further, a weight consolidation scheme is incorporated into the critic update law to attain lifelong learning by overcoming catastrophic forgetting. Finally, a numerical example supports the analytical claims

Optimal Tracking Of Nonlinear Discrete-time Systems Using Zero-Sum Game Formulation And Hybrid Learning

This paper presents a novel hybrid learning-based optimal tracking method to address zero-sum game problems for partially uncertain nonlinear discrete-time systems. An augmented system and its associated discounted cost function are defined to address optimal tracking. Three multi-layer neural networks (NNs) are utilized to approximate the optimal control and the worst-case disturbance inputs, and the value function. The critic weights are tuned using the hybrid technique, whose weights are updated once at the sampling instants and in an iterative manner over finite times within the sampling instants. The proposed hybrid technique helps accelerate the convergence of the approximated value functional to its actual value, which makes the optimal policy attain quicker. A two-layer NN-based actor generates the optimal control input, and its weights are adjusted based on control input errors. Moreover, the concurrent learning method is utilized to ease the requirement of persistent excitation. Further, the Lyapunov method investigates the stability of the closed-loop system. Finally, the proposed method is evaluated on a two-link robot arm and demonstrates promising results

Continual Learning-Based Optimal Output Tracking of Nonlinear Discrete-Time Systems with Constraints: Application to Safe Cargo Transfer

This Paper Addresses a Novel Lifelong Learning (LL)-Based Optimal Output Tracking Control of Uncertain Non-Linear Affine Discrete-Time Systems (DT) with State Constraints. First, to Deal with Optimal Tracking and Reduce the Steady State Error, a Novel Augmented System, Including Tracking Error and its Integral Value and Desired Trajectory, is Proposed. to Guarantee Safety, an Asymmetric Barrier Function (BF) is Incorporated into the Utility Function to Keep the Tracking Error in a Safe Region. Then, an Adaptive Neural Network (NN) Observer is Employed to Estimate the State Vector and the Control Input Matrix of the Uncertain Nonlinear System. Next, an NN-Based Actor-Critic Framework is Utilized to Estimate the Optimal Control Input and the Value Function by using the Estimated State Vector and Control Coefficient Matrix. to Achieve LL for a Multitask Environment in Order to Avoid the Catastrophic Forgetting Issue, the Exponential Weight Velocity Attenuation (EWVA) Scheme is Integrated into the Critic Update Law. Finally, the Proposed Tracker is Applied to a Safe Cargo/ Crew Transfer from a Large Cargo Ship to a Lighter Surface Effect Ship (SES) in Severe Sea Conditions

Optimal Adaptive Tracking Control Of Partially Uncertain Nonlinear Discrete-Time Systems Using Lifelong Hybrid Learning

This article addresses a multilayer neural network (MNN)-based optimal adaptive tracking of partially uncertain nonlinear discrete-time (DT) systems in affine form. By employing an actor–critic neural network (NN) to approximate the value function and optimal control policy, the critic NN is updated via a novel hybrid learning scheme, where its weights are adjusted once at a sampling instant and also in a finite iterative manner within the instants to enhance the convergence rate. Moreover, to deal with the persistency of excitation (PE) condition, a replay buffer is incorporated into the critic update law through concurrent learning. To address the vanishing gradient issue, the actor and critic MNN weights are tuned using control input and temporal difference errors (TDEs), respectively. In addition, a weight consolidation scheme is incorporated into the critic MNN update law to attain lifelong learning and overcome catastrophic forgetting, thus lowering the cumulative cost. The tracking error, and the actor and critic weight estimation errors are shown to be bounded using the Lyapunov analysis. Simulation results using the proposed approach on a two-link robot manipulator show a significant reduction in tracking error by $44\%$ and cumulative cost by $31\%$ in a multitask environment

The Relationship Between Vitamin E Plasma and BAL Concentrations, SOD Activity and Ventilatory Support Measures in Critically Ill Patients

Abstract Vitamin E is a potent reactive oxygen metabolites (ROM) scavenger. It is a lipid-soluble vitamin and its main function is to protect polyunsaturated fatty acids against oxidative stress. Twenty-five mechanically ventilated Intensive Care Unit (ICU) adult patients participated in a prospective randomized clinical trial receiving either placebo (10 patients) or 3 IM doses (1000 IU each) of vitamin E (15 patients). We determined plasma and bronchoalveolar lavage (BAL) fluid concentrations of vitamin E and superoxide dismutase (SOD). Among these 25 patients, there were 14 men and 11 women, aged 63.16 ±15.48 years (mean ± SD; range = 33 to 87 years). Vitamin E supplementation resulted in significant differences in plasma and BAL vitamin E concentrations between the two groups (p-value = 0.01, 0.01), decrease in SOD activities (not differ significantly in plasma (p-value = 0.23)), but with significant differences in BAL (p-value = 0.016) and progressive reduction in Acute Physiology and Chronic Health Evaluation II (APACHE II) (p-value = 0.52) and Sequential Organ Failure Assessment (SOFA) (p-value = 0.008) score in vitamin E group. From the results of this study, it seems that supplementation of vitamin E as a potent antioxidant, along with other supportive measures, can be beneficial in decreasing SOD total activity, ROM production and risk of organ failure in critically ill patients